Abstract Title Page 

Not included in page count. 



Title: Experimental Evaluation of the Tools of the Mind Preschool Curriculum 
Authors and Affiliations: 

Sandra Jo Wilson, Ph.D. 

Senior Research Associate 

Associate Director, Peabody Research Institute 

Vanderbilt University 

230 Appleton Place, PMB 181 

Nashville, TN 37203-5721 

Phone: (615) 343-7215 

email: sandra.j.wilson@vanderbilt.edu 

Dale C. Farran, PhD 

Professor of Education and Psychology 

Department of Teaching and Learning 

Senior Associate Director, Peabody Research Institute 

Vanderbilt University 

230 Appleton Place, PMB 181 

Nashville, TN 37203-5721 

Phone: (615) 343-9515 

email: dale.farran@vanderbilt.edu 



SREE Spring 2012 Conference Abstract Template 




Abstract Body 

Limit 4 pages single-spaced. 



Background / Context: 

This paper will focus on the results of our ongoing experimental evaluation of the Tools of 
the Mind Prekindergarten Curriculum (Bedrova & Leong, 2007), which is being conducted 
in Tennessee and North Carolina. The Tools of the Mind curriculum approach follows from 
socio-cultural perspectives on child development that emphasize how children acquire 
skills and "cultural tools" (e.g., spoken and written language, pretend play, the use of 
numbers, diagrams and maps) in collaboration with knowledgeable others. In the Tools 
approach, the tactics, mediators, forms of talk, and activities the teacher uses to foster 
learning are themselves designed to be part of what the student learns. Tools teaches 
teachers to use dynamic assessment and scaffolding techniques that will help children 
internalize the learning tools, that is, to use the mediators introduced by the teacher and 
then create their own, to apply self-talk and writing, and to use shared activities and 
dramatic play in ways that help them attend, self-monitor, solve problems, plan, and 
remember. 

Purpose / Objective / Research Question / Focus of Study: 

The aim of the Tools of the Mind prekindergarten curriculum is to enhance children’s 
executive function skills within an instructional context that promotes the basic academic 
and social skills that prepare them for kindergarten and beyond. To investigate the 
effectiveness of Tools in achieving this aim, we are conducting a longitudinal randomized 
experiment to answer the following questions: 

1. Do children in Tools of the Mind classrooms improve more in literacy, math, social skills, 
and behavior problems during the preschool year than children in "business as usual" 
control classrooms? Are those gains sustained through kindergarten and first grade? 

2. Do children in Tools of the Mind classrooms show greater gains in executive function 
than children in the control classrooms? Do those gains mediate the curriculum effects 
on literacy, math, and social skills outcomes? 

3. Are there differential effects of Tools of the Mind associated with characteristics of the 
children or the classrooms? 

Setting: 

Four school districts in Tennessee and two in North Carolina are participating in the study. 
The four Tennessee districts and one of the North Carolina districts experienced their test 
year during the 2010-2011 school year; the presentation will focus on these school 
districts. The second North Carolina district is experiencing its test year in 2011-2012. All 
the prekindergarten programs in these schools are funded through grants from their states 
and/or Title 1, thus all families must meet the income guidelines for free or reduced-price 
lunch in order to enroll their children. The 2010-2011 school districts are: 
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1. Lebanon Special School District, an independent district for the city of Lebanon that is 
embedded within the Wilson county district, is located east of Nashville, TN. It serves 
more than 3,000 prekindergarten to 8* grade students in five schools with 25% 
minority and a poverty rate of 51%. It has 5 prekindergarten classrooms in 5 schools. 

2. Wilson County School District is east of Nashville, TN, in a predominantly rural county 
that is experiencing a dramatic increase in the number of Hispanic children. It serves 
more than 13,000 students in prekindergarten to 12* grade with a poverty rate of 
23% and 11% minorities. It has 10 prekindergarten classrooms in 9 schools. 

3. Franklin Special School District serves the city of Franklin and embedded within, but 
independent from, the Williamson County school district. This K-8 school system is 
south of Nashville, TN, and serves 3,900 students in 7 schools. It serves 32% minority 
students with 30% economically disadvantaged and has 7 prekindergarten classrooms 
in 4 schools. 

4. Cannon County Schools is southeast of Nashville, TN, in a predominantly rural, poor 
county. It serves 2,177 children, prekindergarten to 12* grade, the majority of which 
qualify for free or reduced-price lunch (52%) but a low percentage (3.2%) is minority. 
It has 4 prekindergarten classes in 4 schools. 

5. Guilford County School System is an urban system serving 70,000 children in the 
piedmont region of North Carolina. This school system serves a majority of minority 
students (58%), nearly half the students qualify for free or reduced-price lunch (49%). 
It has 77 prekindergarten classes in 45 schools, and 30 of those classrooms in 22 
schools participated in the research. 

Population / Participants / Subjects: 

In all, 847 children from 60 classrooms in 44 schools were seen at the beginning of 
prekindergarten and 801 children at the end of prekindergarten. Demographics for the 
participating children are shown in Table 1 (please insert Table 1 here). Overall, the sample 
of students was diverse in terms of ethnicity and language background, with multiple 
minority groups represented. Close to 30% of the students were English language learners. 

Sixty teachers participated in the study, with 32 in the Tools condition and 28 in the 
comparison condition. Overall, teachers averaged 12 years of teaching experience, with 
seven years in preschool classrooms. All teachers had at least a Bachelor's degree, and over 
half had completed coursework toward or obtained a Master’s degree. In addition, each 
classroom had at least one assistant. 

Intervention / Program / Practice:. 

Tools of the Mind is based on an interactive sequence of change (shown in Figure 1) 
whereby teachers use assessment and scaffolding to tailor their use and modeling of 
specific tactics. These are internalized by their students as cognitive tools, which are then 
used independently and manifested in observable behaviors in the classroom. That set of 
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behaviors we call learning-related self-regulation or executive function. As we describe in 
the data collection section below, our set of outcome measures includes indicators of both 
executive function and other key literacy, language, math, and social & emotional skills 
outcomes. Operationally, Tools of the Mind is both a curriculum and a professional 
development program for teachers. As a curriculum, the focus is on 61 Vygotskian activities 
designed to promote children’s meta-cognitive development. As a part of this evaluation, a 
detailed fidelity of implementation system was devised that tallied the 61 activities 
teachers enacted, the steps they completed, mediators used and the inclusion of incorrect 
actions ("should not’s") as identified by the curriculum developers. 

Research Design: 

This large-scale experimental study was designed to test the effectiveness of the Tools of 
the Mind curriculum when compared to the usual curriculum and practice occurring in the 
participating school systems. Because it is advantageous for the Tools professional 
development if all the prekindergarten teachers within a school are trained together and 
encouraged to support each other during implementation, schools were the unit of 
randomization. This scheme was also intended to minimize interaction between 
experimental and control teachers that might have compromised the comparison. The 
schools were blocked by district, with the large Guilford, NC district divided into two 
blocks. Within each block, half the schools were assigned to the Tools condition and half to 
the practice as usual control condition (with slight variations due to the uneven number of 
schools in some districts). All the prekindergarten classrooms within each school then 
participated in the condition to which the school was assigned. 

The teachers in the classrooms assigned to the control condition continued to practice as 
usual with whatever curriculum they were using, which varied from district to district. The 
teachers in the Tools condition began the professional development sequence for Tools and 
began implementing the Tools curriculum the first year of the study (the 2009-2010 school 
year). However, that first year was a training and practice year for the teachers and no 
measures were taken on the children to assess curriculum effects. The second year (2010- 
2011) was the test year for the classrooms we report on here. 

Data Collection and Analysis: 

We used a battery of child achievement measures as well as a number of direct 
assessments of self-regulation and teacher and assessor behavior rating measures to assess 
the effects of the curriculum. Achievement measures included 7 subtests from the 
Woodcock-Johnson that examine literacy, language, and math skills. The direct assessments 
of self-regulation were selected to capture one or more components of executive function 
including attentiveness, attention shifting, inhibitory control, persistence, and working 
memory. Teachers reported on children’s classroom behavior and language ability. In 
addition, assessors rated children’s self-regulatory behaviors during the assessment 
sessions. 
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Children were consented in both the intervention and comparison classrooms and tested 
on executive function and their academic preparation for kindergarten at the beginning 
and end of preschool. Children were individually assessed by trained and certified 
assessors in two 20-minute sessions. Teachers rated the children’s social skills and 
classroom behavioral competencies in the fall (after 6 weeks of school) and in May. 

Findings / Results: 

The students in the treatment and comparison groups were similar on all demographic 
variables. Furthermore, randomization checks have shown that the treatment and 
comparison groups were similar on all pretest assessments and ratings, with no significant 
differences between the Tools and comparison groups on any measure. 

The effectiveness of the Tools curriculum was tested using multi-level regression models 
with students nested within classrooms, schools, and district blocks. The models for each 
outcome included pretest scores, age, interval between assessments, gender, ELL status, 
and ethnicity as covariates. 

Our results show that there were no significant treatment effects on any of our outcome 
variables. Students in Tools classrooms performed about equally well on all outcome 
variables, including the executive function measures, after receiving a year of the 
curriculum as students who received the usual preschool curriculum. Similarly there were 
no differences between the two sets of classrooms in teacher ratings of social and 
behavioral competence. In addition, the Tools curriculum did not appear to result in 
significantly better outcomes for any student subgroups (i.e., ELL, ethnic groups, gender) 
when compared to the control condition. Gains on all outcomes were observed across the 
preschool year in both Tools and comparison classrooms. 

The presentation will summarize and report on the statistical models tested, examine the 
effects of the curriculum for demographic and regional subgroups of students. Descriptives 
for the main outcomes are shown in Table 2 (please insert Table 2 here). 

Conclusions: 

Given the widespread interest and growing adoption of the Tools of the Mind curriculum, 
the curriculum developers and research team at the Peabody Research Institute agreed 
that a rigorous experimental evaluation of the curriculum was necessary. While analyses 
thus far have not shown significant treatment effects, results from the Kindergarten and 
future grade assessments might evidence results that appear late as the cognitive 
demands of schooling increase. Furthermore, our results have not shown the curriculum to 
be any less effective than the curricula used in the comparison classrooms. Further 
analyses are being conducted to examine classroom processes in Tools and comparison 
classrooms more closely to investigate the theoretical model of the Tools approach. 
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Table 1. Demographics 


Variable 


Tools 




Comparison 




n 


% 


n 


% 


Male 


262 


53% 


217 


57% 


Female 


237 


47% 


162 


43% 




ELL 


137 


28% 


110 


29% 


Not ELL 


362 


72% 


269 


71% 



Black/African American 


145 


29% 


86 


23% 


Hispanic/Latino 


119 


24% 


95 


25% 


Caucasian 


192 


38% 


157 


41% 


Other 


43 


9% 


41 


11% 




Age at pretest 


4.5 years 


54m 


4.6 years 


55m 


Age at posttest 


5.2 years 


62m 


5.2 years 


62m 
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Table 2. Pretest and Posttest Descriptives 
on Academic Outcomes (W scores) and Self-Regulation Direct Assessments 







Tools 




Comparison 






Mean 


sd 


n 


Mean 


sd 


n 


Letter Word Pretest 


314.8 


25.8 


493 


312.6 


26.2 


369 


Letter Word Posttest 


347.0 


22.6 


465 


348.8 


23.3 


348 


Spelling Pretest 


337.7 


23.3 


493 


335.0 


23.4 


369 


Spelling Posttest 


372.8 


25.4 


465 


370.7 


26.9 


348 


Academic Knowledge Pretest 


427.2 


23.4 


493 


426.6 


22.9 


369 


Academic Knowledge Posttest 


443.5 


17.1 


465 


443.0 


18.3 


348 


Oral Comprehension Pretest 


439.4 


16.1 


493 


438.8 


15.7 


369 


Oral Comprehension Posttest 


450.1 


16.2 


465 


450.4 


17.2 


348 


Picture Vocabulary Pretest 


450.9 


25.4 


493 


451.1 


24.3 


369 


Picture Vocabulary Posttest 


462.4 


15.3 


465 


463.2 


15.0 


348 


Applied Problems Pretest 


381.1 


31.8 


493 


380.6 


30.8 


369 


Applied Problems Posttest 


407.6 


21.2 


465 


407.0 


22.5 


348 


Quantitative Concepts Pretest 


403.6 


12.9 


493 


402.4 


13.0 


369 


Quantitative Concepts Posttest 


421.3 


15.1 


465 


421.3 


14.7 


348 


Dimensional Change Card Sort 
Pretest 


1.3 


0.6 


493 


1.3 


0.6 


371 


Dimensional Change Card Sort 
Posttest 


1.7 


0.6 


465 


1.6 


0.6 


348 


Forward Digit Span Pretest 


2.5 


1.3 


493 


2.5 


1.3 


370 


Forward Digit Span Posttest 


3.1 


1.2 


465 


3.1 


1.1 


348 


Backward Digit Span Pretest 


1.2 


1.2 


493 


1.2 


1.1 


369 


Backward Digit Span Posttest 


1.6 


1.3 


465 


1.6 


1.4 


348 


Peg Tapping Pretest 


4.4 


5.8 


493 


4.3 


5.8 


368 


Peg Tapping Posttest 


9.4 


5.6 


465 


9.2 


6.0 


348 


Head-Toes-Knees-Shoulders 

Pretest 


10.5 


13.6 


492 


9.6 


12.2 


366 


Head-Toes-Knees-Shoulders 

Posttest 


22.4 


17.2 


464 


21.2 


17.1 


348 


Copy Design Pretest 


1.1 


1.6 


492 


1.0 


1.5 


368 


Copy Design Posttest 


5.2 


2.8 


465 


4.8 


2.8 


348 
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